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EXECUTIVE  SUMMARY 


Goals 

The  principal  goals  of  this  research  project  are  to:  (a)  discover  the 
nature  of  judgment  and  decision  making  In  dynamic  tasks;  (b)  study  the  effects 
of  stress  on  judgment  and  decision  making  in  such  tasks.  Neither  project  has 
been  undertaken  previously  by  researchers  in  this  field. 


Initial  Plan  of  Work 

We  had  intended  to  (a)  study  college  sophomores  and  others  in  laboratory 
situations  (Year  I),  (b)  move  to  the  study  of  professional  experts  in  dynamic 
tasks  (Year  II)  and  (c)  study  experts  under  stress  in  dynamic  tasks  (Year 
III).  This  plan  was  changed  midway  in  the  first  year  when  it  became  apparent 
that  we  would  have  an  excellent--and  rare--opportunity  to  move  directly  to  the 
situation  planned  for  Year  II;  that  is,  to  the  study  of  weather  forecasters 
coping  with  dynamic  tasks  (forecasting  microbursts,  hall,  and  severe  weather) 
with  a  clear  possibility  of  working  with  these  same  experts  (and  same  tasks) 
under  stressful  conditions,  thus  providing  continuity  in  moving  us  directly 
toward  our  principal  goals.  Because  of  the  recognition  of  the  importance  of 
research  of  this  sort,  the  Center  for  Atmospheric  Research  (NCAR)  and  the 
National  Oceanic  and  Atmospheric  Administration  (NOAA)  not  only  offered  us  the 
cooperation  of  their  expert  forecasters,  but  offered  supplemental  funding  for 
the  ARI  project.  As  a  result  we  were  able  to  employ  additional  staff  members 
(an  expert  Ph.D.  level  judgment  researcher  and  a  research  assistant).  As  our 
Quarterly  Reports  have  indicated,  we  moved  quickly  to  take  advantage  of  these 
unusual  circumstances  and  midway  in  the  first  year  began  work  planned  to  begin 
in  Year  II,  that  is,  direct  study  of  experts  coping  with  dynamic  tasks. 


Current  Efforts 

We  have  undertaken  three  projects  (the  study  of  meteorologists'  efforts 
to  forecast  microbursts,  hail,  and  severe  weather).  Each  is  described  in 
turn. 
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Microburst  Forecasting 

The  invitation  to  participate  in  this  project  (sponsored  by  the  Federal 
Aviation  Administration)  offers  the  unusual  opportunity  to  study  experts  who 
are  attempting  to  (a)  understand  a  type  of  dynamic  environmental  activity 
(weather  phenomena  of  critical  importance  to  air  travel)  and  (b)  forecast  its 
occurrence.  Moreover,  the  meteorologists  who  are  undertaking  this  work  are 
provided  with  the  most  sophisticated  state-of-the-art  information  gathering 
and  display  devices  available  (e.g.,  Doppler  radar),  thus  providing  us  with 
laboratory  conditions  ordinarily  unimaginable.  And  because  of  the  full 
cooperation  of  the  project  manager  and  staff  meteorologists,  we  have  been  able 
to  make  field  observations  as  well  as  laboratory- type  studies  of  these 
experts'  efforts  to  understand  and  predict  the  behavior  of  complex  dynamic 
systems.  Each  effort  is  described  in  turn. 


Fi el d  observations.  During  these  observations  we  acquainted  ourselves 
with  the  complicated  computer  hardware  and  software  available  to  the 
meteorologists,  and  observed  the  meteorologists'  activities  under  real  time 
conditions.  These  observations  led  us  to  the  conclusion  that  several  sources 
of  information  and  several  display  modes  are  available  to  the  meteorologists. 
In  these  conditions,  the  meteorologists  (a)  are  active  seekers  of  Information 
that  changes  over  time,  (b)  are  active  hypothesis  generators  about  what  has 
happened.  Is  happening,  and  will  happen,  (c)  employ  both  deductive  and 
inductive  inferences,  (d)  rely  partially  on  scientific  principles  and 
partially  on  empirical  regularities  and  thus  employ  both  functional  analyses 
and  pattern  matching  strategies,  and  (e)  make  judgments  and  decisions  about 
dynamic  environmental  events  under  considerable  uncertainty  that  is  created  by 
Incomplete  and  Imprecise  data,  and  incomplete  physical,  chemical,  and 
climatological  principles.  In  our  view,  this  situation  Is  far  from  those 
ordinarily  studied  in  laboratory  situations  by  psychologists,  but  highly 
representative  of  the  vast  majority  of  conditions  under  which  judgments  and 
decisions,  military  and  otherwise,  are  made.  This  situation,  therefore,  is 
Ideal  for  the  present  research  project.  We  also  concluded  that  our  studies 
must  follow  from  a  new  and  much  more  complex  psychological  theory  of  judgment 
and  decision  making  than  has  thus  far  been  advanced  by  anyone. 


Laboratory-type  studies.  Our  first  steps  were  to  discover  to  what  extent 
the  experts  would  exhibit  inter-  and  intra-observer  agreement  with  regard  to 
the  forecasting  of  microbursts  when  given  (a)  perfectly  reliable  measurements 
of  weather  data  (assumption  that  observers  of  the  radar  screen  and  other 
Instruments  should  show  perfect  inter-  and  intra-observer  agreement)  and  (b) 
perfectly  reliable  (as  observed)  interpretations  of  those  data,  as  for 
example,  Interpreting  the  wind  velocities  to  signify  such  microburst 
precursors  as  "descending  core,  collapsing  storm,  etc.  Only  modest  agreement 
among  the  experts  was  found.  This  result  (Including  the  finding  that  one 
meterologlst's  judgments  were  far  different  from  the  others)  was  new 
Information  to  the  forecasters.  We  then  investigated  the  question  of  whether 
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the  forecasters  could  translate  profile  information  about  storms  (as  described 
above)  into  mental  images  of  the  storms,  and  to  what  extent  the  storm  images 
would  be  similar  across  forecasters.  Although  it  was  found  that  the 
forecasters  could  easily  draw  schematic  diagrams  of  storms  depicted  in  the 
profiles,  only  modest  agreement  among  the  forecasters  was  found  with  regard  to 
the  similarity  of  storm  images.  And  when  asked  to  transform  their  storm 
images  back  into  profiles  only  a  rough  translation  appeared  possible  (though 
analysis  is  continuing  on  this  question).  A  Technical  Report  on  the  above 
project  is  in  preparation. 


Hail 


This  project  differs  from  the  microburst  project  in  that  actual  data 
including  a  criterion  for  evaluating  the  accuracy  of  forecasters  were 
available.  Cue  data  (from  Doppler  radar  scans  of  thunderstorms)  and  criterion 
data  (observation  of  hail  by  chase  teams  on  the  ground)  were  provided  by  NOAA 
for  453  cases.  Since  an  expert  system  for  hail  prediction  has  been  developed 
by  NOAA,  it  was  possible  to  set  up  a  competition  among  forecasters'  forecasts, 
the  AI  expert  system's  forecasts,  and  forecasts  based  on  judgment  models  of 
the  forecasters.  It  was  found  that  the  three  methods  of  forecasting  produced 
roughly  the  same  accuracy  (rather  low,  because  of  the  inherent  uncertainty  in 
predicting  hall  from  the  available  data).  The  judgment  model,  however,  is  by 
far  the  least  costly  and  simplest  to  employ,  and  thus  the  most  efficient.  We 
know  of  only  one  previous  study  that  made  such  comparisons;  similar  results 
were  found.  Work  on  this  project  continues.  The  analysis  of  the  data  are 
nearly  complete  and  a  Technical  Report  on  this  project  is  in  preparation. 


Severe  Weather 


This  study  examines  weather  forecasters'  judgments  in  a  dynamic  task  of 
critical  Importance.  NOAA  provided  access  to  an  advanced  Interactive  computer 
graphics  weather  forecasting  workstation  for  this  experiment,  thus  providing 
us  with  a  laboratory  of  great  complexity  and  representations.  Real  weather 
data  were  replayed  in  simulated  forecasting  sessions.  Three  forecasters, 
working  Individually,  were  asked  to  think  aloud  as  they  decided  when  and  where 
to  Issue  warnings  and  alerts  for  thunderstorms.  Their  deliberations  were  tape 
recorded  and  a  record  was  kept  of  their  concurrent  actions.  Comparison  of 
severe  weather  warnings  and  alerts  Issued  by  different  forecasters  for  the 
same  day  Indicate  that  forecasters  generally  focus  on  the  same  storms  (of  the 
several  apparent),  but  the  actions  taken  with  regard  to  those  storms  vary 
substantially  among  forecasters.  Transcripts  of  the  forecasters'  verbal 
descriptions  of  their  forecasting  strategies  are  being  analyzed.  They  suggest 
that  differences  among  forecasters  are  due  to  differences  In  (a)  selection  of 
information,  (b)  strategies  for  integrating  Information,  (c)  Implicit  criteria 
or  thresholds  for  Issuing  warnings  or  alerts.  A  Technical  Report  on  this 
project  is  in  preparation. 
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Theoretical  and  Conceptual  Work 

Our  study  of  weather  forecasters  making  judgments  in  dynamic  tasks  under 
actual  conditions  where  instrumentation  of  considerable  complexity  is  used  has 
led  to  a  definite  conclusion  about  present  theory  in  judgment  and  decision 
research,  namely,  it  is  far  too  simple  to  be  of  use  in  dynamic  tasks  of  any 
reasonable  complexity.  Therefore  we  are  developing  a  general  theory  to 
account  for  the  cognitive  activity  of  these  experts  under  these  complex 
conditions.  A  Technical  Report  is  in  preparation. 


